Data retrieval based on storage device activation schedules

ABSTRACT

In at least one embodiment, a method of operating a multiple-data-storage-devices enclosure is disclosed. The method includes: receiving a first read request from a first requester device for a first requested data fragment; identifying a first target data storage device storing the first requested data fragment based at least partly on the first read request; activating, independently of receiving the first read request, only a subset of data storage devices in the enclosure, wherein the subset includes the first target data storage device; retrieving, based at least partly on the first read request and in response to activating the subset, the first requested data fragment from the first target data storage device; and transmitting the first requested data fragment to the first requester device.

RELATED FIELD

This disclosure relates generally to a data storage system (also,“storage system”), and in particular to archival storage systems forinfrequently accessed data.

BACKGROUND

Commercial enterprises (e.g., companies) and others gather, store, andanalyze an increasing amount of data. The trend now is to store andarchive almost all data before making a decision on whether or not toanalyze the stored data. Although the per unit costs associated withstoring data has declined over time, the total costs for storage hasincreased for many companies because of the volumes of stored data.Hence, it is important for companies to find cost-effective ways tomanage their data storage environments for storing and managing largequantities of data. Companies now manage these costs by having varioustiers of storage, with different costs associated with each of thesetiers. Each tier can have different data storage hardware (e.g., storageprocessor, storage medium, storage I/O network, etc.) and differentstorage services (e.g., data maintenance, data integrity check, backuppoint-in-time images, etc.). Companies can use the different tiers ofstorage for different types of data. As an example, to store data thatis accessed frequently, companies may use a data storage tier that hashigh performance characteristics. On the other hand, for big dataapplications, companies often prefer high density and/or high storagevolume archival storage systems, which tend to be less expensive on aper unit basis. However, it is often a challenge to keep the cost downfor these archival storage systems due to the necessity of maintaining alarge number of data storage devices. To drive the cost of archivalstorage systems down while maintaining high storage density,conventional solutions tend to sacrifice computing capabilities, e.g.,by removing storage efficiency services and/or reducing storage accessbandwidth.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a perspective plan view of a multiple-data-storage-devicescartridge and components therein, consistent with various embodiments.

FIG. 1B is a backside view of the multiple-data-storage-devicescartridge of FIG. 1A, consistent with various embodiments.

FIG. 1C is a perspective view of a rack of multiple-data-storage-devicescartridges, consistent with various embodiments.

FIG. 2 is a block diagram of a multiple-data-storage-devices cartridge,consistent with various embodiments.

FIG. 3 is a block diagram illustrating a system architecture of anarchival storage system, consistent with various embodiments.

FIG. 4 is a control flow diagram of the archival storage system,consistent with various embodiments.

FIG. 5 is a control flow diagram of a storage front-end system,consistent with various embodiments.

FIG. 6 is a flow chart of a process of synchronized deduplication acrossa spin group, consistent with various embodiments.

FIG. 7 is a flow chart of a process of synchronized erasure codingacross a spin group, consistent with various embodiments.

FIG. 8 is a control flow of an archival storage system capable ofuncoordinated data access queuing, consistent with various embodiments.

FIG. 9 is a flow chart of a process to monitor lifespan consumption ofdata storage devices in a multiple-data-storage devices cartridge,consistent with various embodiments.

FIG. 10 is a flow chart of a process for predictive fault sensing ofdata storage devices in a multiple-data-storage devices cartridge,consistent with various embodiments.

FIG. 11 is a flow chart of a process of token-based cascade staging ofmultiple-data-storage devices cartridges, consistent with variousembodiments.

FIG. 12 is a block diagram illustrating a diagrammatic representation ofa computing device within which a set of instructions, for causing themachine to perform any one or more of the methodologies or modulesdiscussed herein, may be executed.

The figures depict various embodiments of the disclosure for purposes ofillustration only. One skilled in the art will readily recognize fromthe following discussion that alternative embodiments of the structuresand methods illustrated herein may be employed without departing fromthe principles of the disclosure described herein.

DETAILED DESCRIPTION

Disclosed are embodiments of an archival data storage system implementedby multiple-data-storage-devices cartridges. Eachmultiple-data-storage-devices cartridge (“cartridge”) is an enclosure oftwo or more data storage devices (e.g., hard disk drives, solid statedrives, hybrid drives, etc.), in which a subset of the data storagedevices (e.g., a single data storage device) can be supplied power andaccessed at a time. Presently, hard disk drives offer low per-unitcosts, but it is possible that other types of data storage devices mayeventually be cheaper than hard disk drives. The archival storage systemis designed to store data that is rarely accessed (“cold” data). To savecost, the data storage devices used inside the cartridges are storagedevices that may have a limited lifetime. For example, each data storagedevice in a cartridge may be designed to read data more frequently thanto write data. In some embodiments, data are only be permitted to bewritten on each data storage devices once. In some embodiments, at leastsome of the data storage devices may be designed to fail after anaverage of around ten or so read end-to-end read accesses. The reducedlifetime is designed by using lower quality components, e.g., read headsas compared to the write heads in the data storage devices.Lower-quality components generally lead to lower costs, so when usedappropriately, they can lower the total solution cost.

The data storage devices can be other types of low-cost, archival-classdisk drives or other high-density data storage devices. The archivalstorage system is designed for cold data that is generally written onlyonce to a data storage device and read back a few times, but notfrequently. In some embodiments, one can estimate the limited lifetimeof a data storage device by the number of write operations and thenumber of read operations (e.g., estimating based on a total number ofoperations or estimating based on the maximum of either the number ofwrite operations or the number of read operations).

The archival storage system implements a data range applicationprogramming interface (API), e.g., through a number of data rangeprocessor modules. A data range processor module is a run-time serviceprovided by one or more cartridges via an executable program implementedby a data processing component (e.g., a low power processor) of thecartridge. In some embodiments, the low-power processor may be anARM-type processor or other processor generally deployed in presentlycommercialized cellular telephones and/or other low-power mobilecomputing devices. The data range APIs enable the archival storagesystem to scale in capacity, e.g., by adding more data storage devicesper cartridge, and in both capacity and data throughput, e.g., by addingmore cartridges. Data throughput may be increased by adding cartridgesbecause each data range module is able to function in parallel to oneanother. For example, if an archival storage system with a single rackof cartridges can be completely filled in 15 days, a larger deploymentof a dozen racks of cartridges can still be filled within 15 days due tothe parallel nature of each data range module.

FIG. 1A is a perspective plan view of a multiple-data-storage-devicescartridge 100 (“the cartridge 100”) and components therein, consistentwith various embodiments. The cartridge 100 includes an enclosure shell102 (partially shown) that encloses and protects multiple data storagedevices 104. The data storage devices 104 may be hard drives,solid-state drives, flash drives, tape drives, or any combinationthereof. It is noted that the term “enclose” does not necessarilyrequire sealing the enclosure and does not necessarily requireenveloping all sides of the enclosure.

The cartridge 100 further includes control circuitry 106 that managesthe power supply of the cartridge 100, the data access to and from thedata storage devices 104, and other storage operations to the datastorage devices 104. The control circuitry 106 may implement each of itsfunctions as a single component or a combination of separate components.

As shown, the cartridge 100 is adapted as a rectangular prism that sitson an elongated surface 108 of the rectangular prism. Each of the datastorage devices 104 may be stacked within the cartridge 100. Forexample, the data storage devices 104 can stack on top of one anotherinto columns. The control circuitry 106 can stack on top of one or moreof the data storage devices 104 and one or more of the data storagedevices 104 can also stack on top of the control circuitry 106.

In various embodiments, the enclosure shell 102 encloses the datastorage devices 104 without providing window openings to accessindividual data storage devices or individual columns of data storagedevices. In these embodiments, each of the cartridge 100 is disposablesuch that after a specified number of the data storage devices 104 fail,the entire cartridge can be replaced as a whole instead of replacingindividual failed data storage devices. Alternatively, the cartridge 100may be replaced after a specified time, e.g., corresponding to anexpected lifetime.

The illustrated stacking of the data storage devices 104 in thecartridge 100 enables a higher density of standard disk drives (e.g.,3.5 inch disk drives) in a standard shelf (e.g., a 19 inch width rackshelf). Each of the cartridge 100 can store ten of the standard diskdrives. In the cases that the data storage devices 104 are disk drives,the cartridge 100A can hold the disk drives “flat” such that the axes ofthe spinning disks are parallel to the gravitational field.

The cartridge 100 may include a handle 110 on one end of the enclosureshell 102 and a data connection port 112 (not shown) on the other end.The handle 110 is attached on an outer surface of the enclosure shell102 to facilitate carrying of the cartridge 100. The enclosure shell 102exposes the handle 110 on its front surface. For example, the handle 110may be a retractable handle that retracts to fit next to the frontsurface when not in use.

FIG. 1B is a backside view of the multiple-data-storage-devicescartridge 100, consistent with various embodiments. For example, thebackside view can be taken from a perspective along A-A′ as shown inFIG. 1A. The enclosure shell 102 encloses the data storage devices 104(not shown) and the control circuitry 106 (not shown), and exposes adata connection port 112, e.g., an Ethernet port. The data connectionport 112 is a component attached to the control circuitry 106. The dataconnection port 112 is an interconnection interface that accepts orplugs into a communication medium, e.g., Ethernet cable. The dataconnection port 112 may be the only component within the cartridge 100exposed outside the enclosure shell 102. For example, the dataconnection port 112 can accept an Ethernet cable carrying communicationsignals and power in accordance with the Power-over-Ethernet (PoE)standards (e.g., IEEE 802.3af-2003 or IEEE 802.3at-2009).

FIG. 1C is a perspective view of a rack 150 ofmultiple-data-storage-devices cartridges, consistent with variousembodiments. The multiple-data-storage-devices cartridges may beinstances of the cartridge 100 illustrated in FIG. 1A. The rack 150, asillustrated, includes a tray structure 152 (e.g., a rack shelf) securingfour instances of the cartridge 100. The tray structure 152 can be astandard 2U 19″ deep rack mount. The rack 150 may include a stack oftray structures 152, each securely attached to a set of rails 162.Management devices 164 may be placed at the top shelves of the rack 150.For example, the management devices 164 may include network switches,power regulators, front-end storage appliances, or any combinationthereof.

FIG. 2 is a block diagram of a multiple-data-storage-devices cartridge200, in accordance with various embodiments. Themultiple-data-storage-devices cartridge 200 (the “cartridge 200”) can bethe cartridge 100 of FIG. 1A. The cartridge 200 includes a dataprocessing module 202, an operational memory 206, a boot flash 208, adata communication port 210, a power management module 212, storageinterfaces 214, and data storage devices 216.

The data processing module 202 can be a microprocessor, a controller, anapplication-specific integrated circuit, a field programmable gatearray, or any combination thereof. The boot flash 208 is a memory devicestoring an operating system 218. The data processing module 202 can loadthe operating system 218 into the operational memory 206 and run theoperating system 218. A data access application programming interface(API) service 220 can execute on this operating system to provide dataaccess over a network to the data storage devices 216 for clients (e.g.,devices, applications, or systems).

The data communication port 210 enables the cartridge 200 to connectwith the network. For example, the data communication port 210 can be aPower-over-Ethernet module that connects to an Ethernet cable to bothestablish a network connection with the network and power the cartridge200.

In various embodiments, the cartridge 200 only turns on a subset(hereinafter the “active set”) of data storage devices 216 at a time.The active set can be a single data storage device or more than one datastorage devices. The data access API service 220 can determine themembership of the active set depending on client requests receivedthrough the network. A client can either specifically request access toa data storage device or request a data range for the data access APIservice 220 to determine which data storage device stores the datarange.

The power management module 212 provides electronic circuitry to switchon and off components of the cartridge 200, e.g., to activate only onesubset of the data storage devices at a time. The power managementmodule 212 can receive instructions from the data processing module 202(e.g., as part of the data access API service 220) to provide power tothe designated active set, including a subset of the storage interfaces214 that enables data access to the active set. Once power is suppliedto the designated active set, the storage controller 222 can facilitatecommunicate between the data processing module 202 through the storageinterface 214 to the data storage devices.

FIG. 3 is a block diagram illustrating a system architecture of anarchival storage system 300, consistent with various embodiments. Thearchival storage system 300 includes multiple data range processormodules 302, each managing multiple data storage devices 304 (e.g., thedata storage devices 104 of FIG. 1A or the data storage devices 216 ofFIG. 2). Each data range processor module 302 is implemented in amultiple-data-storage-devices enclosure 306. As shown, each data rangeprocessor module 302 activates only one active storage device 308 at atime, where the active storage device 308 is selected by the data rangeprocessor module 302 from the data storage devices 304.

Each data range processor module 302 can be implemented as a data accessAPI service (e.g., the data access API service 220 of FIG. 2) running onan operating system implemented by a data processing component (e.g.,the data processing module 202 of FIG. 2) of themultiple-data-storage-devices enclosure 306 (e.g., the cartridge 100 ofFIG. 1A or the cartridge 200 of FIG. 2). Each data range processormodule 302 operates in parallel to one another. The data access APIservice may be implemented under the Representational State Transfer(REST) architecture as a “RESTful http” service. For example, the APIservice can include an interface to authenticate a client, to respond toa request for a list of data ranges, to identify supporting capabilitiesof the API service, to retrieve a data range, and to retrieve metadataabout a data range.

The consumer of the API service provided by the data range processormodules 302 can either be a front-end network manager 310 or one ofclients 312 (e.g., client 312A, client 312B, and client 312C,collectively as the “clients 312”). The front-end network manager 310manages access to the multiple data range processor modules 302. Thefront-end network manager 310 receives data access requests from theclients 312, including write requests and read requests. For each dataaccess request from the clients 312, the front-end network manager 310can sub-divide the data access request into sub-requests, includingwrite sub-requests for portions of payload data indicated in a masterwrite request and read sub-requests for portions of a requested datasetindicated in a master read request.

When writing data, the front-end network manager 310 or one of theclients 312 sends a directed, broadcast/multicast or fanout message toone or multiple data range processor modules 302 announcing an intent tostore data and the nature of the request, e.g., the size of the payloaddata to be written. The nature of the write request may also include anydata processing or data maintenance requests associated with the writerequest. Each of the data range processor modules 302 then determineswhether to respond to the received message. This determination may bebased at least partly on whether the responding data range processormodule has an operable data storage device capable of storing acontiguous range of data matching the size of the payload data. That is,whether one of the active or inactive data storage devices managed bythe responding data range processor module has sufficient availablememory space for the size of the payload data.

In some cases, multiple data range processor modules respond to thebroadcast/multicast message. The front-end network manager 310 or therequesting client can select which of the respondent modules (i.e.,those data range processor modules that responded) are to receive thepayload data. One or more of respondent modules may be selected, eacheither to store the entire payload data as mirrors of one another or tostore a divided or encoded portion of the payload data. The front-endnetwork manager 310 or the requesting client can select the respondentmodules based at least partly on the order of when each respondentmodule responded to the directed, broadcast/multicast, or fanoutmessage, a load-balancing factor associated with the respondent module,amount of capacity available to the respondent module, age of therespondent module, likelihood of failure of the respondent module,importance of the payload data, a geolocation of the respondent module,or any combination thereof.

After the front-end network manager 310 or the requesting client selectswhich of the respondent modules to send the payload data or portions ofthe payload data, the front-end network manager 310 or the requestingclient delivers the payload data or the portions of the payload data tothe selected respondent modules. The delivering of the payload data orthe portions of the payload data may be in response to selection of therespondent modules. At least some of the selected respondent modules laydown the payload data or a received portion of the payload data on theactive storage device 308 as a range (i.e., contiguous data).

When laying down the range, each data range processor module 302 doesnot go back and modify data already written. All data is written onlyonce to the data storage devices 304. In various embodiments, incomingranges are laid down starting from a last written point on the activestorage device 308. If the respondent module has no record of a lastwritten point, sector zero of the active storage device 308 is set asthe starting point.

Each data range processor module 302 can maintain a range identifiertable that associates the active storage device 308 with the writtendata ranges. The range identifier table can later be used to locate thewritten data ranges. The data ranges can also be written in aself-describing data layout. For example, as a data range is written tothe active storage device 308, a range identifier and size/length of thedata range is also written (e.g., as a header or a footer). Each of thedata range processor modules 302 lays down data ranges in its activestorage device 308 sequentially. Later, when attempting to recover whatis written on a data storage device, a skip-seek mechanism can be usedto scan across the data storage device by looking for the headers orfooters of the data layout to reconstruct the range identifier table.Under this data layout, the archival storage system 300 does not supportmodification or deletion of the data ranges. Likewise, when reading adata range from the active storage device 308, a data range processormodule streams the data range to the requesting party sequentially (asopposed to block-based access).

In the disclosed architecture, the front-end network manager 310 and theclients 312 are API consumers of the data access API services providedby the multiple data range processor modules 302. The front-end networkmanager 310 and the clients 312 are unaware which data storage devicesare activated versus deactivated/powered down. Under this architecture,the data range processor modules 302 function independently of oneanother.

From an architectural standpoint, the overall storage capacity of thearchival storage system 300 can be scaled by adding more data storagedevices 304 to each of the multiple-data-storage-devices enclosure 306.The overall throughput of the archival storage system 300 can be scaledby adding more of the multiple data storage devices enclosures.Consequently, the overall storage capacity of the archival storagesystem 300 is also increased when additional multiple data storagedevices enclosures are added.

In some embodiments, the front-end network manager 310 or the clients312 can store a metadata structure mapping data ranges to the multipledata range processor modules 302. For example, responsive to receiving aread request for a data range from one of the clients 312, the front-endnetwork manager 310 can first select which of the data range processormodules to contact via the metadata structure, and then request the datarange from the selected data range processor module(s). The metadatastructure can also include an indication of the specific data storagedevices managed by the data range processor modules 302 that are storingthe data ranges.

In such cases, a range request to the selected data range processormodule(s) also indicates to the selected data range processor module(s)which data storage device(s) to activate. Alternatively, the data rangeprocessor modules 302 can store their own metadata structure mappingdata ranges to the multiple data storage devices 304. In such cases,each of the selected data range processor module(s) can activate aspecific data storage device or subset of data storage devices inresponse to the range request and retrieve the requested data range forthe front-end network manager 310.

In other embodiments, when retrieving a specific data range, thefront-end network manager 310 or the clients 312 can send abroadcast/multicast message querying the data range processor modules302 for the specific data range, e.g., using a range identifier orlabel. In these embodiments, the data range processor modules 302 keeptrack of which data ranges they have (e.g., through a table of datarange identifiers and/or hash values) and the corresponding data storagedevices storing the data ranges. A data range processor module that isaware of the specific data range can then activate a corresponding datastorage device containing the specific data range, retrieve the specificdata range, and respond to the broadcast/multicast message with theretrieved data range.

In various embodiments, multiple data range processor modules storingthe same fragment of data can respond to a broadcast/multicast readrequest. The front-end network manager 310 can take the first copy itreceives to process the retrieval of the requested data object.

In various embodiments, the multiple data range processor modules 302not only service external clients and/or the front-end network manager310, but also perform data storage device management services, includingdata storage device health monitoring, data storage devicedecommissioning, data recovery, failure recovery, or any combinationthereof. The multiple data range processor modules 302 can also maintaindata management services, including updating the metadata structures ofdata ranges.

The data range processor modules 302 do not support deletion ormodification of data ranges. If the clients 312 send a request to“delete” or “modify” a data range, a data range processor module canmark the data range as “gone” or “deleted.” The data range processormodule does not reclaim this memory space while the active storagedevice 308 is in operation. Instead, if the active storage device 308has enough data ranges marked as “deleted,” the remaining data ismigrated off the active storage device 308 to be preserved, and theactive storage device 308 is wiped clean. Subsequently, the activestorage device 308 can be reused as an empty data storage device.

FIG. 4 is a control flow diagram of an archival storage system 400,consistent with various embodiments. The archival storage system 400includes a front-end subsystem 402 (e.g., the front-end network manager310 of FIG. 3) and multiple storage subsystems 404 (e.g., the cartridge100 of FIG. 1A, the cartridge 200 of FIG. 2, or themultiple-data-storage-devices enclosure 306 of FIG. 3). The front-endsubsystem 402 can be one or more computer systems (e.g. the computingdevice 1200 of FIG. 12), having either a shared nothing architecture ora shared database architecture, connected to the storage subsystems 404over a network (e.g., a global network or a local network). Thefront-end subsystem 402 can be on a separate rack from the storagesubsystems 404.

The front-end subsystem 402 includes a protocol interfaces module 406.The protocol interfaces module 406 defines one or more functionalinterfaces that applications and devices used to store, retrieve,update, and delete data elements from the archival storage system 400.For example, the protocol interfaces module 406 can implement a CloudData Management Interface (CDMI), a Simple Storage Service (S3)interface, or both. While the archival storage system 400 is adapted tonever delete or update any written data, the archival storage system 400can emulate a deletion of data ranges by removing metadata associatedwith the data ranges and/or marking an indication that the memory spacesoccupied by the data ranges are available for overwrite. The archivalstorage system 400 can emulate an update by storing an updated datarange as a new data range, updating metadata associated to the updateddata range, and marking any metadata associated of the old data rangewith an indication of deletion.

The front-end subsystem 402 also includes a staging area 408. Thestaging area 408 is a memory space implemented by one or more datastorage devices within or accessible to the front-end subsystem 402. Forexample, the staging area 408 can be implemented by solid-state drives,hard disks, volatile memory, or any combination thereof. The stagingarea 408 can maintain a file namespace 410 to facilitate clientinteractions through the protocol interfaces module 406. The filenamespace 410 manages a set of data container identifiers, eachcorresponding to a dataset from clients of the front-end subsystem 402.The staging area 408 also maintains a fragment namespace 412corresponding to the file namespace 410. The fragment namespace 412manages a set of fragment identifiers, each corresponding to a datarange stored in the multiple storage subsystems 404. The staging area408 can store a mapping structure 414 that stores associations betweenthe data container identifiers of the file namespace 410 and thefragment identifiers of the fragment namespace 412.

In some embodiments, the front-end subsystem 402 can be implemented as adistributed computing network including multiple computing nodes (e.g.,computer servers). Each computing node can include an instance of thestaging area 408. The namespaces (e.g., the file namespace 410 and thefragment namespace 412) of each staging area 408 can be implementedeither as a share-nothing database or a shared database.

The staging area 408 can also serve as a temporary cache to processpayload data from a write request received at the protocol interfacesmodule 406. The front-end subsystem 402 can process incoming writerequests via a data processing pipeline through a pipeline plannermodule 416. When the pipeline planner module 416 receives an incomingwrite request, the pipeline planner module 416 can determine what typesof storage efficiency processes to perform on the payload data of thewrite request prior to sending the payload data into persistent storagein the storage subsystems 404. The storage efficiency processes includecompression of the payload data as implemented by a compression module418, deduplication of the payload data as implemented by a deduplicationbinning module 420 and a deduplication module 422, fragment encryptionas implemented by a fragment encryption module 424, and erasure codingas implemented by a fragment erasure coding module 426. The pipelineplanner module 416 can improve storage efficiency by allocatingprocessing time (e.g., as measured by absolute time, processing cycles,or process iterations) to each of these storage efficiency processes.While the storage efficiency processes is illustrated in FIG. 4 in agiven order, alternative embodiments may perform these storageefficiency processes in a different order, and some processes may beremoved, moved, added, subdivided, combined, and/or modified to providealternatives or subcombinations. The scheduled order of storageefficiency processes can be saved to enable a reversal of the storageefficiency processes in order to execute a subsequently received clientread request to retrieve the payload data.

The compression module 418 implements one or more compressionalgorithms. The pipeline planner module 416 can schedule the compressionmodule 418 to try a subset or all of the one or more compressionalgorithms on the payload data. The compressed payload data taking upthe least memory space can then be piped to the next storage efficiencyprocess.

The deduplication binning module 420 implements one or morededuplication binning techniques. The term “deduplication binning” hererefers to a method of splitting data chunks (e.g., the payload datachunks or the compressed payload data chunks) into data fragments(“bins”) and performing deduplication against a chosen fragment ofchunks. The deduplication module 422 implements one or morededuplication techniques working in conjunction with the one or morededuplication binning techniques. The pipeline planner module 416 canschedule for any combination of deduplication binning techniques anddeduplication techniques on the payload data or the compressed payloaddata. The deduplicated payload data from a combination of deduplicationbinning and deduplication techniques with the best result (e.g., largestamount of redundant data removed) can then be piped to the next storageefficiency process.

The fragment encryption module 424 implements one or more encryptiontechniques. The pipeline planner module 416 can schedule the fragmentencryption module 424 to encrypt fragments of the incoming data (e.g.,deduplicated and compressed payload data). As part of the deduplicationbinning process, the payload data can be split into data fragments. Thefragment encryption module 424 can encrypt the resultant fragments ofthe payload data from the deduplication process. The encrypted fragmentsare then piped to the next storage efficiency process.

The fragment erasure coding module 426 implements one or more erasurecoding techniques. The term “erasure coding” herein refers totransforming payload data of k fragments into erasure coded data of Nfragments, where N>k, such that the payload data can be recovered from asubset of the N. The pipeline planner module 416 can schedule thefragment erasure coding module 426 to try a subset or all of the one ormore erasure coding techniques on the incoming fragments (e.g.,deduplicated fragments and/or encrypted fragments). As part of thededuplication binning process or the fragment encryption process, thepayload data can be split into data fragments. The fragment erasurecoding module 426 can erasure code the resultant fragments of thepayload data from the deduplication process and/or the fragmentencryption process. The erasure coded fragments with the optimal coderate and reception efficiency are then piped to the next storageefficiency process. The term “code rate” herein refers to the fractionk/N and the term “reception efficiency” refers to the fraction k′/k,where k′ denotes the minimum number of fragments needed for recovery ofthe payload data.

FIG. 5 is a control flow diagram of a storage front-end system 500,consistent with various embodiments. The storage front-end system 500may be a front-end subsystem (e.g., the front-end subsystem 402) for anarchival storage system, e.g., the archival storage system 300 of FIG. 3or the archival storage system 400 of FIG. 4. The storage front-endsystem 500 includes one or more control interfaces 502 and one or moreprotocol interfaces 504. The control interfaces 502 can providemonitoring, management configuration and other non-data-relatedfunctions. The protocol interfaces 504 feed into the file/objectnamespace module 506 (e.g., the file namespace 410 of FIG. 4). Theprotocol interfaces 504 can be the Cloud Data Management Interface(CDMI) interface or the S3 interface. The protocol interfaces 504 enablethe storage front-end system 500 to interact with clients over anetwork. The protocol interfaces 504 can deliver client requestsincluding both read requests and write requests to the file objectnamespace module 506. A client request can include an object identifierand a service level objective (SLO) of the request.

If the client request is a write request, the file object namespacemodule 506 adds the write request (e.g., including a data objectidentifier and a write request SLO) to a write queue 508. The payload ofthe write request can be stored in a file object staging area 510 (e.g.,the staging area 408 of FIG. 4). The write queue 508 can process writerequests cached therein in the order the write requests are received(absent a message that overwrites a priority of one of the writerequests), or process write requests out of order. The write queue 508can process each write request through a write planner module 512 (e.g.,the pipeline planner module 416 of FIG. 4). Based on the payload dataand other information in the write request (e.g., the SLO), the writeplanner module 512 selects storage preprocessing techniques that areused to process the write request. The write planner module 512 can alsodetermine an ordering of how the storage preprocessing techniques are toapply on the payload data and parameters for running the storagepreprocessing techniques. The write planner module 512 can transmit theselection of techniques, the ordering of applying the techniques, andthe parameters for the techniques in a transformation recipe to astorage preprocessor subsystem 514. The write planner module 512 canoperate iteratively. That is, the write planner module 512 can plan totry a permutation of techniques and/or options for the techniques, andbased on the tested result, further changes the options or thetechniques to try to optimize one or more variables (e.g., storageperformance indicators) to achieve a better end result.

The storage preprocessor subsystem 514 implements the storagepreprocessing techniques. For example, the storage preprocessorsubsystem 514 can include a deduplication module 516 (e.g., thededuplication module 422 and the deduplication binning module 420 ofFIG. 4), a compression module 518 (e.g., the compression module 418 ofFIG. 4), a fragmentation module 520 to divide the payload data intofragments (e.g., for deduplication binning, erasure coding, and/orfragment encryption), an erasure coding module 522 (e.g., the fragmenterasure coding 426 of FIG. 4), an encryption module 524 (e.g., thefragment encryption module 424 of FIG. 4), or any combination thereof.The storage preprocessor subsystem 514 can also include one or more ofthe compression module 418 of FIG. 4, the deduplication binning module420 of FIG. 4, the deduplication module 422 of FIG. 4, the fragmentencryption module 424 of FIG. 4, and the fragment erasure coding module426 of FIG. 4.

After processing the payload data in accordance with the chosentransformation recipe, the storage preprocessor subsystem 514 depositsthe processed fragments into a fragment staging area 526. Thetransformation recipe for the processed fragments stored as part of oneor more of the fragments, and can also be stored into a metadata storage528. The metadata storage 528 stores both the transformation recipe andoptionally a data layout of the processed fragments. The signal pathlabeled “purge” allows the storage front-end system 500 to reclaim spacein the File/Object Namespace 506 when the data is fully stored asfragments.

A fragment namespace module 530 can retrieve the processed fragmentsfrom the fragment staging area 526 and store the processed fragments toone or more repositories. For example, the repositories can include tapedrives, cloud storage, local or external disk drives, the cartridge 100of FIG. 1A, the cartridge 200 of FIG. 2, themultiple-data-storage-devices enclosure 306 of FIG. 3, the storagesubsystems 404 of FIG. 4, or any combination thereof. The fragmentnamespace module 530 can store the data layout of the processedfragments in the metadata storage 528.

If the client request is a read request, the file object namespacemodule 506 adds the read request (e.g., including a requested dataobject identifier and a read request SLO) to a read queue 540. The readqueue 540 can process read requests cached therein in the order the readrequests are received (absent a message that overwrites a priority ofone of the read requests), or process the reads out of order. The readqueue 540 can process each read request through a read planner module542. Based on the data object identifier and other information in theread request (e.g., the SLO), the read planner module 542 can retrieve atransformation recipe corresponding to the requested data object fromthe metadata storage 528 or from a corresponding fragment.

The read planner module 542 can send a fragment retrieval request to thefragment namespace module 530. In response, the fragment namespacemodule 530 can retrieve the fragments corresponding to the requesteddata object from the repositories. The fragment namespace module 530 canidentify the data layout of the fragments corresponding to the requesteddata object from the metadata storage 528.

The read planner module 542 can then send the retrieved fragments to areverse processor subsystem 544 to reconstruct the requested data objectfrom the retrieved fragments. Based on the transformation recipe, theread planner module 542 can select reverse processing techniques, anordering of how the reverse processing techniques are to apply on theretrieved fragments, and parameters for running the reverse processingtechniques. The read planner module 542 can send the retrieved fragmentsalong with the selection of techniques, the ordering of the techniques,and the parameters of the techniques.

The reverse processor subsystem 544 may include a decryption module 546,an erasure decoding module 548, a reassembly module 550, a decompressionmodule 552, an un-deduplication module 554, or any combination thereof.The decryption module 546 can apply one or more decryption techniques onthe fragments, each technique utilizing a cryptographic key. Thedecryption module 546 can be configured in association with theencryption module 524. For example, the decryption module 546 and theencryption module 524 can share a symmetric key or can each possess anasymmetric key coupled to one another (e.g. a private decryption key anda public encryption key). The erasure decoding module 548 canreconstruct an original set of fragments utilizing all or a subset oferasure coded fragments. The erasure decoding module 548 can beconfigured in association with the erasure coding module 522 such thatthe erasure decoding module 548 can reverse the effects of the erasurecoding module 522.

The reassembly module 550 can assemble the de-processed fragments backto a continuous dataset. The reassembly module 550 can be configured inassociation with the fragmentation module 520 to reverse the effects ofthe fragmentation module 520. The decompression module 552 can executeone or more decompression techniques to expand a compressed dataset toits raw format. The decompression module 552 can be configured inassociation with the compression module 518 to reverse the effects ofthe compression module 518. The un-deduplication module 554 can beconfigured in association with the deduplication module 516 to reversethe effects of the deduplication module 516.

After the reverse processor subsystem 544 reconstructs the original dataobject, the reverse processor subsystem 544 deposits the original dataobject in the file/object staging area 510. The original data object canbe a file, an object, a volume, a data range sequence, a binary string,a data aggregate, or any combination thereof. The file/object namespacemodule 506 can determine when the original data object is deposited intothe file object staging area 510. In response, the file/object namespacemodule 506 can respond to the read request via at least one of theprotocol interfaces 504 by sending the original data object back to theclient.

Implementations of the storage preprocessor subsystem 514 and thereverse processor subsystem 544 enable the storage front-end system 500to improve storage efficiency using storage processing pipelineoptimization. The storage preprocessor subsystem 514 implements apipeline of storage preprocessing techniques that improves storageefficiency. The storage preprocessor subsystem 514 presumes that therepositories utilized by the fragment namespace module 530 arehigh-latency storage devices, e.g., the multiple-data-storage-devicesenclosure 306 where storage devices therein are frequently deactivated.Because of this, the storage preprocessor subsystem 514 utilizes theadditional time to optimize the pipeline for even higher storageefficiency that traditional systems could not previously achieve.

Regarding FIGS. 1-5, portions of components and/or modules associatedwith the cartridge 100, the cartridge 200, the archival storage system300, the archival storage system 400, and the storage front-end system500 may each be implemented in the form of special-purpose circuitry, orin the form of one or more appropriately programmed programmableprocessors, or a combination thereof. For example, the modules describedcan be implemented as instructions on a tangible storage memory capableof being executed by a processor or a controller in the controlcircuitry. The tangible storage memory may be volatile or non-volatilememory. In some embodiments, the volatile memory may be considered“non-transitory” in the sense that it is not a transitory signal.Modules may be operable when executed by a processor or other computingdevice, e.g., a single board chip, an application specific integratedcircuit, a field programmable field array, a network capable computingdevice, a virtual machine terminal device, a cloud-based computingterminal device, or any combination thereof. Memory spaces and storagesdescribed in the figures can be implemented with the tangible storagememory as well, including volatile or non-volatile memory.

Each of the modules and/or components may operate individually andindependently of other modules or components. Some or all of the modulesmay be executed on the same host device or on separate devices. Theseparate devices can be coupled together through one or morecommunication channels (e.g., wireless or wired channel) to coordinatetheir operations. Some or all of the components and/or modules may becombined as one component or module.

A single component or module may be divided into sub-modules orsub-components, each sub-module or sub-component performing a separatemethod step or method steps of the single module or component. In someembodiments, at least some of the modules and/or components share accessto a memory space. For example, one module or component may access dataaccessed by or transformed by another module or component. The modulesor components may be considered “coupled” to one another if they share aphysical connection or a virtual connection, directly or indirectly,enabling data accessed or modified from one module or component to beaccessed in another module or component. In some embodiments, at leastsome of the modules can be upgraded or modified remotely. The cartridge100, the cartridge 200, the archival storage system 300, the archivalstorage system 400, or the storage front-end system 500 may includeadditional, fewer, or different modules for various applications.

At least one embodiment involves a method of operating a storagefront-end manager system to perform pipeline planning for a high latencystorage system. The method can include: receiving a write requestincluding payload data; storing the payload data of the write request ina staging area of the storage front-end manager system; determining atransformation pipeline based at least partly on an attribute of thewrite request; queuing the transformation pipeline for execution on thepayload data to generate data fragments for storage; executing thetransformation pipeline; and transmitting the data fragments to aplurality of multiple-data-storage-devices enclosures after thetransformation pipeline is executed.

The staging area can be a cache memory with at least a data objectnamespace and a fragment namespace. The method can further includeassociating an object identifier of the payload data in the data objectnamespace and fragment identifiers of the data fragments in the fragmentnamespace. The method can also further include storing a layout of thedata fragments, where the layout indicates which of the plurality ofmultiple-data-storage-devices enclosures stores each of the datafragments. The layout can also include an indication of which datastorage device(s) within each of the plurality ofmultiple-data-storage-devices stores each of the data fragments. Themethod can also include tracking a group of data storage devices in theplurality of multiple-data-storage-devices enclosures and associatingthe group with the object identifier for the payload data.

The attribute of the write request may include a service level objective(SLO) of the write request. When determining the transformationpipeline, the storage front-end manager system can determine thetransformation pipeline based at least partly on the SLO.

Determination of the transformation pipeline can include selecting acompression technique to process the payload data, selecting afragmentation technique to fragment the payload data, selecting adeduplication technique to reduce redundancy in the payload data,selecting an encryption technique to protect the payload data fromunauthorized access, and/or an erasure coding technique so that afraction of the data fragments can reconstruct the payload data. Thefragmentation technique can be selected based at least partly on adeduplication binning requirement, an erasure coding requirement, or acombination thereof.

Executing the transformation pipeline can include executing a pluralityof storage efficiency steps sequentially and piping an output of eachstorage efficiency step as an input to a subsequent storage efficiencystep to transform the payload data through the transformation pipeline.Determination of the transformation pipeline can also include selectingmultiple trial techniques to optimize storage efficiency of a firststorage efficiency step of the storage efficiency steps. Executing thetransformation pipeline can include executing the multiple trialtechniques and selecting one of the multiple trial techniques totransform the payload data based on a measurable metric of how well eachof the multiple trial techniques performed in optimizing storageefficiency. The storage front-end manager system can store atransformation recipe of the transformation pipeline in a transformationdatabase. The transformation recipe can include at least transformationtechniques used in the storage efficiency steps, an order of the storageefficiency steps, parameters of the transformation techniques, or anycombination thereof.

At least one embodiment involves a method of operating a storagefront-end manager system to read data. The method can include: receivinga read request including a data object identifier associated with arequested data object; retrieving a transformation recipe associatedwith the data object identifier and a fragment layout associated withthe data object identifier; generating a reverse transformation pipelinebased at least partly on the transformation recipe; retrieving datafragments from a plurality of data storage devices based at least partlyon the fragment layout; queuing the reverse transformation pipeline forexecution on the data fragments to reconstruct the requested dataobject; and executing the reverse transformation pipeline.

The storage front-end manager system can retrieve the data fragments bytransmitting direct, broadcast or multicast, or fanout message, based atleast partly on the fragment layout, to request the data fragments froma plurality of multiple-data-storage-devices enclosures and receivingthe data fragments from a subset of the plurality ofmultiple-data-storage-devices enclosures. The storage front-end managersystem can receive the data fragments from the subset by asynchronouslywaiting to receive each of the data fragments from eachmultiple-data-storage-devices enclosure in the subset. That is, themultiple-data-storage-devices enclosures may each include a subset ofthe plurality of data storage devices containing the data fragments, andmay activate the subset based on its own schedule.

FIG. 6 is a flow chart of a process 600 of synchronized deduplicationacross a spin group, consistent with various embodiments. The process600 begins with a storage front-end system assigning, at block 602, datastorage devices across two or more multiple-data-storage-devicescartridges (e.g., the cartridge 100 of FIG. 1A, the cartridge 200 ofFIG. 2, or the multiple-data-storage-devices enclosure 306 of FIG. 3) asa synchronized spin group. The multiple-data-storage-devices cartridges(the “cartridges”) are enclosures containing multiple data storagedevices, where each enclosure turns on only a subset of its multipledata storage devices at a time. The storage front-end system can be thefront-end subsystem 402 of FIG. 4 or the storage front-end system 500 ofFIG. 5. The membership of the synchronized spin group can be stored inthe front-end subsystem 402. Each cartridge can be independent of eachother, and thus, does not have any knowledge of its data storage deviceshaving memberships in particular synchronized spin groups.

A “spin group” referred to in this disclosure does not imply that thedata storage devices within the group necessarily have the ability tospin, such as a hard disk, but rather the member data storage devicesare intended, at times, to function as a whole and be activatedtogether. Data stored across the data storage devices of the spin groupcan be used to reconstruct client data that was processed by the storagefront-end system together. In various parts of the disclosure, a “spingroup” may be referred to as a “synchronous group” of data storagedevices. Under some conditions, the synchronous group can be activatedand/or deactivated together by a storage front-end system. Under otherconditions, multiple-data-storage-devices enclosures containing membersof the same synchronous group can maintain data fragments associatedwith each other, without necessarily activating the member data storagedevices at the same time.

Once assigned together, the cartridges can activate the data storagedevices in the synchronized spin group together at the same time. Forexample, the storage front-end system can send an instructionsimultaneously to the multiple-data-storage-devices cartridgescontaining the member data storage devices to activate the member datastorage devices immediately. The cartridges can also deactivate the datastorage devices in the synchronized spin group together. Memory spacewithin the synchronized spin group can be presented as a storage“aggregate,” a single logical data container across several physicaldata containers (e.g., data storage devices).

The membership of the synchronized spin group can follow a numericalpattern. For example, the storage front-end system can assign everyfirst disk drive of a set of cartridges as belonging to a firstsynchronized spin group, every second disk drive of the set ofcartridges as belonging to a second synchronized spin group, and etc.This configuration reduces the complexity requirements of managing thememberships of the synchronized spin groups on the storage front-endsystem.

At block 604, the storage front-end system can initiate a deduplicationtask that is targeted to access at least a portion of the storage“aggregate” corresponding to the synchronized spin group. Block 604 canbe in response to the storage front-end system receiving a client'swrite request or in response to meeting a deduplication schedule.Responsive to initiating the deduplication task, the storage front-endsystem can send a message to the cartridges containing members of thesynchronized spin group to activate the member data storage devices atblock 606. The message can instruct the cartridges to immediatelyactivate the member data storage devices. Alternatively, responsive tothe initializing, the deduplication task can be stored in a queue towait until the synchronized spin group is activated together by acyclical schedule. For example, the storage front-end system canperiodically activate different spin groups in a set sequence.

Once the member data storage devices in the synchronized spin group areactivated, the storage front-end system can perform, at block 608, thededuplication task across the storage aggregate formed by thesynchronized spin group. Deduplication is a process of eliminatingduplicate copies of repeating data within a data container, such as thestorage aggregate.

The disclosed technique of deduplicating synchronized spin groupsenables deduplication of an expandable aggregate of data storage deviceseven though the data storage devices are stored in different cartridgesand sometimes deactivated (e.g., to save storage maintenance oroperational costs). Because the spin group can be activated together, astorage system implementing the process 600 can perform low latencydeduplication across a large number of cartridges.

The external management of the membership of the synchronized spin groupenables the individual cartridges containing the member data storagedevices to remain independent of each other. This architecture canremove or reduce increases in complexity (e.g., complexity associatedwith maintaining cluster membership information) when scaling the numberof cartridges in the archival storage system. While the process 600 isdirected at performing a deduplication task across the synchronized spingroup, the same process may also be directed at other data maintenancetasks as well.

Some embodiments involve a method of operating a storage front-endsystem. The method can include: identifying a deduplication group ofdata storage devices across two or more multiple-data-storage-devicesenclosures, wherein data across the data storage devices is accessibleby the storage front-end system as an aggregate memory space;transmitting a request to the two or more multiple-data-storage-devicesenclosures to near immediately activate the data storage devices; anddeduplicating at least a portion of the data in the aggregate memoryspace by accessing via a data communication network the data storagedevices.

The method can further include receiving a write request with payloaddata, wherein the identifying of the deduplication group can be inresponse to receiving the write request and the deduplicating caninclude deduplicating the payload data across the aggregate memoryspace. Alternatively, the identifying of the deduplication group can bein response to meeting a data maintenance schedule.

Transmitting of the request to activate can be directed at a superset ofmultiple-data-storage-devices enclosures that includes the two or moremultiple-data-storage-devices enclosures. In order for the two or moremultiple-data-storage-devices enclosures to recognize that the requestis for them, the request can include a group identifier or data storagedevice identifiers of the deduplication group.

The method can further include determining when the two or moremultiple-data-storage-devices enclosures have activated the data storagedevices. The deduplicating of the at least a portion of the data canoccur after determining that the data storage devices have beenactivated. The deduplication group can be identified by accessing agroup identity storage in the storage front-end system. The groupidentity storage can include a group identifier associated withidentifiers of the data storage devices. The group identifier can alsobe associated with identifiers of the two or moremultiple-data-storage-devices enclosures. The storage front-end systemcan associate the group identifier with the identifiers of the datastorage devices or the multiple-data-storage-devices enclosures whenprocessing deduplication binning of payload data from a write request.

The method can further include initiating a data operation directed at amember data storage device of the deduplication group. In response toinitiating the data operation, the storage front-end manger system canrequest the two or more multiple-data-storage-devices enclosures toactivate the deduplication group immediately.

The data storage devices in the deduplication group can share a commonattribute across the two or more multiple-data-storage-devicesenclosures. For example, the common attribute can be a commonenumeration according to the two or more multiple-data-storage-devicesenclosures (e.g., each of the data storage devices is the 1st datastorage device in its respective enclosure).

Other embodiments involve a different method of operating a storagefront-end system. For example, the method can include initiating a datamaintenance task that requires access to at least a first data storagedevice; identifying a synchronous group of data storage devices acrosstwo or more multiple-data-storage-devices enclosures that the first datastorage device is associated with (i.e., belongs in), wherein dataacross the data storage devices is accessible by the storage front-endsystem as an aggregate memory space; sending an activation request tothe two or more multiple-data-storage-devices enclosures to nearlyimmediately activate the data storage devices in the synchronous group;and performing the data maintenance task in the aggregate memory spaceof the data storage devices. The data maintenance task can be adeduplication process. The data maintenance task can be associated witha read request or a write request.

The activation request can be sent to a superset ofmultiple-data-storage-devices enclosures that includes the two or moremultiple-data-storage-devices enclosures. The activation request canidentify the synchronous group of the data storage devices. The methodcan further include determining when the data storage devices in thesynchronous group have been activated. The storage front-end system canthen perform the data maintenance task after the data storage deviceshave been activated.

Performing the data maintenance task can include requesting datafragments from the data storage devices in the synchronous group, andreceiving and caching the data fragments from the two or moremultiple-data-storage-devices enclosures. The data maintenance task canthen be performed based at least in part on the data fragments.

In various embodiments, these methods of performing a deduplication taskor a data maintenance task associated with the synchronous group of thedata storage devices can be performed by a computer system serving as astorage front-end system (e.g., the computer device 1200 of FIG. 12).

FIG. 7 is a flow chart of a process 700 of synchronized erasure codingacross a spin group, consistent with various embodiments. The process700 begins with a storage front-end system assigning, at block 702, datastorage devices across two or more multiple-data-storage-devicescartridges (e.g., the cartridge 100 of FIG. 1A, the cartridge 200 ofFIG. 2, or the multiple-data-storage-devices enclosure 306 of FIG. 3) asa synchronized spin group based at least partly on a write request. Insome embodiments, the synchronized spin group here can be used by thestorage front-end system as the synchronized spin group discussed inFIG. 6.

The multiple-data-storage-devices cartridges (the “cartridges”) areenclosures containing multiple data storage devices, where eachenclosure turns on only a subset of its multiple data storage devices ata time. The storage front-end system can be the front-end subsystem 402of FIG. 4 or the storage front-end system 500 of FIG. 5. The membershipof the synchronized spin group can be stored in the front-end subsystem402. Each cartridge can be independent of one another and thus does nothave any knowledge of its data storage devices having memberships inparticular synchronized spin groups.

Once assigned together, the cartridges can activate the data storagedevices in the synchronized spin group together at the same time. Forexample, the storage front-end system can send an instruction to themultiple-data-storage-devices cartridges containing the member datastorage devices to activate the member data storage devices immediately.The cartridges can also deactivate the data storage devices in thesynchronized spin group together.

The membership of the synchronized spin group can follow a numericalpattern. For example, the storage front-end system can assign everyfirst disk drive of a set of cartridges as belonging to a firstsynchronized spin group, every second disk drive of the set ofcartridges as belonging to a second synchronized spin group, and etc.This configuration reduces the complexity requirements of managing thememberships of synchronized spin groups in the storage front-end system.

At block 704, the storage front-end system can perform erasure coding onthe payload data of the write request to generate data fragments. Only asubset of the data fragments are needed to reconstruct the data object.Block 704 can be in response to the storage front-end system receiving aclient write request. At block 706, the storage front-end system canstore the data fragments individually in different member data storagedevices of the synchronized spin group. At block 708, the storagefront-end system can store an association between the payload data andthe synchronized spin group (e.g., by associating a data objectidentifier of the payload data with a group identifier of thesynchronized spin group).

A storage front-end system implementing the process 700 can activateand/or deactivate the synchronized spin group together in response tofuture read requests. The external management of the membership of thesynchronized spin group enables the individual cartridges containing themember data storage devices to remain independent of each other.

For example, at block 710, the storage front-end system can send arequest to the cartridges to immediately activate the data storagedevices in the synchronized spin group in response to a read request forthe data object. Subsequently, the storage front-end system can receiveat least a subset of the data fragments from a subset of thesynchronized spin group at block 712. Then at block 714, the storagefront-end system can erasure decode the subset of data fragments toreconstruct a contiguous data range representing the original payloaddata/data object.

When the member data storage devices of the synchronized spin groups arehard disks and/or solid-state drives, the synchronized spin groupsenable unique access latency characteristics that is advantageous overtraditional systems. While waiting for a synchronized spin group toactivate, any storage operations to the storage aggregate of the spingroup would experience high latency. However, after the synchronizedspin group activates, the actual performance of the storage aggregatehas low latency. This enables complex storage operations (e.g.,deduplication or erasure coding) that span across a large storagecapacity to execute under low latency on a low-cost archival storagesystem.

Some embodiments involve a method of operating a storage front-endsystem to handle a read request. The method can include: receiving theread request with an object identifier for a data object; identifying asynchronous group of data storage devices across two or moremultiple-data-storage-devices enclosures, wherein the synchronous groupis associated with the object identifier; sending a request (e.g., adirect, network broadcast or multicast, or fanout message) to the two ormore multiple-data-storage-devices enclosures to immediately activatethe data storage devices in the synchronous group; retrieving at least asubset of data fragments associated with the object identifier from atleast one of the data storage devices; and erasure decoding at least thesubset of the data fragments into a contiguous data range to reconstructthe data object. The method can further include retrieving atransformation pipeline associated with the object identifier andtransforming the contiguous data range into the data object based atleast partly on the transformation pipeline. For example, thetransforming the contiguous data range can include decompression,decryption, un-deduplication, or a combination thereof, of thecontiguous data range.

The request can be sent to a superset of multiple-data-storage-devicesenclosures that includes the two or more multiple-data-storage-devicesenclosures. The request can identify at least the synchronous group andthe data fragments such that the two or moremultiple-data-storage-devices enclosures can recognize that they need toactivate their respective data storage devices.

Part of sending the request can include identifying the data storagedevices. For example, the data storage devices can be identified byaccessing identifiers of the data storage devices associated with thesynchronous group for the data object. Subsequently, the storagefront-end system can include the identifiers of the data storage devicesin the request. As another example, the storage front-end system canaccess identifiers of the two or more multiple-data-storage-devicesenclosures associated with the synchronous group for the data object.The identifiers of the two or more multiple-data-storage-devicesenclosures can then be included in the request.

The data storage devices in the synchronous group share a commonattribute across the two or more multiple-data-storage-devicesenclosures similar to the deduplication group discussed above. Thecommon attribute can be a common enumeration according to the two ormore multiple-data-storage-devices enclosures.

The storage front-end system can receive just enough of the datafragments in the subset that are sufficient to erasure decode andreconstruct the contiguous data range. The storage front-end system candetermine how many data fragments are sufficient to erasure decode byaccessing a transformation recipe associated with the object identifier.For example, a write request comes in, the storage front-end system canselect a transformation pipeline that utilizes a specific erasure codingtechnique. The erasure coding technique can have a reception efficiencyassociated therewith that governs how portion of the stored datafragments is needed to construct the original payload data of the writerequest. During the processing of a write request, the storage front-endsystem can store the reception efficiency information associated with aobject identifier in the transformation recipe.

Other embodiments involve a method of operating a storage front-endsystem to handle a write request. The method can include: receiving thewrite request including payload data; processing the payload data forstorage in response to the write request, wherein the processingincludes erasure-coding the payload data or partially processed payloaddata into data fragments; identifying a synchronous group of datastorage devices across two or more multiple-data-storage-devicesenclosures; storing the data fragments across the data storage devicesof the identified synchronous group; and associating the payload datawith a group identifier of the synchronous group. Associating thepayload data with the group identifier can include associating an objectidentifier of the payload data with the group identifier in a dataobject namespace of the storage front-end system.

The storage front-end system can identify the synchronous group byselecting the two or more multiple-data-storage-devices enclosures froma superset of multiple-data-storage-devices enclosures; and assigning atleast a data storage device from each of the two or moremultiple-data-storage-devices enclosures as part of the synchronousgroup. The selection may be based on information regarding the datastorage devices in the two or more multiple-data-storage-devicesenclosures. For example, the storage filesystem can send an inquiry tothe superset regarding available storage space and/or failure riskfactor of data storage devices in the respectivemultiple-data-storage-devices enclosures. Selecting of the two or moremultiple-data-storage-devices enclosures can be based at least partly onresponses to the inquiry from the enclosures. Alternatively, instead ofcreating a new synchronous group, identifying the synchronous group caninclude selecting the synchronous group from pre-existing synchronousgroups whose identities are stored in the storage front-end system.

In various embodiments, these methods of performing a data retrievaltask or a data storage task associated with the synchronous group of thedata storage devices can be performed by a computer system serving as astorage front-end system (e.g., the computer device 1200 of FIG. 12).

FIG. 8 is a control flow of an archival storage system 800 capable ofuncoordinated data access queuing, consistent with various embodiments.The archival storage system 800 can be the archival storage system 300of FIG. 3 or the archival storage system 400 of FIG. 4. The archivalstorage system 800 may be accessed by clients (e.g., client 802A andclient 802B shown as dashed line boxes, collectively the “clients 802”).The archival storage system 800 includes multiple front-end subsystems(e.g., a front-end subsystem 804A and a front-end subsystem 804B,collectively the “front-end subsystems 804”) that process clientrequests (e.g., a read request 806A and a read request 806B,collectively the “read requests 806”). The front-end subsystems 804 mayeach include a request queue (e.g., a request queue 808A of thefront-end subsystem 804A and a request queue 808B of the front-endsubsystem 804B, collectively the “request queues 808”).

In the illustrated example, the read requests 806A and the read request806B target a data object in the archival storage system 800. Each ofthe clients 802 may be directed to an arbitrary front-end subsystem(e.g., by the client's own choosing or by a network device of thearchival storage system 800). For example, the read request 806A can bedirected to the front-end subsystem 804A and the read request 806B canbe directed to the front-end subsystem 804B. In response to receivingthe read requests 806, the front-end subsystem 804A queues the readrequest 806A in the request queue 808A and the front-end subsystem 804Bqueues the read request 806B in the request queue 808B.

The front-end subsystems 804 can independently communicate, via direct,broadcast or multicast, or fanout message, its respective read requestto two or more multiple-data-storage-devices cartridges (e.g., acartridge 810A, a cartridge 810B, and a cartridge 810C, collectively the“cartridges 810”) in storage enclosure bank 812. The storage enclosurebank 812 can include a number of spin groups (e.g., a spin group 814A, aspin group 814B, a spin group 814C, a spin group 814D, and a spin group814E, collectively the “spin groups 814”), each having one or moremember data storage devices that contain fragments of a user data set.The front-end subsystems can determine and store the memberships of thespin groups 814 when writing user data to the cartridges 810. The readrequest communicated to the cartridges 810 may also include identifiersof the member data storage devices.

Under the disclosed architecture, the cartridges 810 do not have toactivate and/or deactivate the member data storage devices at the sametime as each other. For example, the cartridge 810A has a drive “B”activated, the cartridge 810B has a drive “A” activated, and thecartridge 810C has a drive “D” activated. Each cartridge 810 can followa schedule of activation and deactivation of its data storage devices815. Each cartridge 810 can optimize the schedule to maximize thelifespan of the data storage devices 815. Optionally, the schedule canbe delayed or changed in response to a cartridge receiving a commandfrom one of the front-end subsystems 804 to activate a specific datastorage device. Also optionally, when a front-end subsystem establishesa spin group, the front-end subsystem can publish a recommended scheduleof activation and/or deactivation to every cartridge that contains amember of that spin group.

For example, the spin group 814A may be represented by disk driveslabeled as “A” in each of the cartridges 810. The spin group 812A maycontain the data object requested by the read requests 806. Eachcartridge that receives the read requests 806 can queue the readrequests 806 in its response queue 816. When a member data storagedevice of the spin group 814A activates on one of the cartridges 810,one or more fragments of the requested data object are sent back to thefront-end subsystems 804 (i.e., to satisfy the read requests from thefront-end subsystem 804A and the front-end subsystem 804B,respectively).

The front-end subsystems 804 can implement a shared nothingarchitecture. The disclosed architecture enables uncoordinated retrievalof any data object with fragments stored on data storage devices acrossany number of storage enclosures (e.g., the cartridges 810). At no pointduring the process of data object retrieval do the front-end subsystems804 have to coordinate with each other. This architecture enablesscaling the client service throughput without needing to maintain alarge database or a global scheduler. This architecture for the archivalstorage system 800 also enables the client service throughput to scaleindependently of the storage capacity.

Some embodiments involve a method of operating amultiple-data-storage-devices enclosure (the “enclosure”) to performuncoordinated data retrieval. The method can include: receiving a firstread request from a first requester device for a first requested datafragment; identifying a first target data storage device storing thefirst requested data fragment based at least partly on the first readrequest; activating, independently of receiving the first read request,only a subset of data storage devices in the enclosure, wherein thesubset includes the first target data storage device; retrieving, basedat least partly on the first read request and in response to activatingthe subset, the first requested data fragment from the first target datastorage device; and transmitting the first requested data fragment tothe first requester device. The first read request can also be directedat other multiple-data-storage-devices enclosures.

The first read request can include a storage device identifier of thefirst target data storage device. In this case, the enclosure canidentify the first target data storage device based on the storagedevice identifier. Alternatively, the first read request can include afragment identifier of the first requested data fragment or an objectidentifier of a data object that originated the first requested datafragment. In this case, the enclosure can identify the first target datastorage device based on the fragment identifier or the object identifierand a mapping of the fragment identifier or the object identifier to thefirst target data storage device. The mapping can be stored in a memoryof the enclosure. In yet another alternative embodiment, the first readrequest includes a group identifier associated with multiple datastorage devices across two or more multiple-data-storage-devicesenclosures. In this case, the enclosure can identify the first targetdata storage device based on the group identifier and a mapping of thegroup identifier to the first target data storage device. Similarly,this mapping can be stored in a memory of the enclosure.

The method described above can also further include storing the firstread request in a request queue; and waiting to dequeue the first readrequest from the request queue to initiate the retrieving of the firstdata fragment until the first target data storage device is activated.The request queue can store read requests and/or write requests specificfor the first target data storage device. In some embodiments, therequest queue is dequeued in a first in first out manner. In otherembodiments, dequeuing of the request queue is not restricted to thefirst in first out manner.

The method described above can also process a second read requestconcurrent to the first read request. For example, the method canfurther include: receiving the second read request from a secondrequester device for a second requested data fragment after receivingthe first read request; identifying a second target data storage devicestoring the second requested data fragment based at least partly on thesecond read request; activating the second target data storage devicebefore, concurrently or after the first target data storage device;retrieving, based at least partly on the second read request and inresponse to activating the second target data storage device, the secondrequested data fragment from the second target data storage device; andtransmitting the second requested data fragment to the second requesterdevice before, concurrently or after transmitting the first requesteddata fragment.

Activating the subset can be part of a data storage device activationcycle (e.g., a preset cycle) that rotates through different subsets ofthe data storage devices in the enclosure. Alternatively, the enclosurecan determine an activation schedule and the activating of the subsetcan be in accordance with the activation schedule. The activationschedule can be determined based at least partly on health statistics ofthe data storage devices in the enclosure.

Some other embodiments involve a method of operating a storage front-endsystem to facilitate the uncoordinated data retrieval across themultiple-data-storage-devices enclosures. The method can include:receiving a read request that includes an object identifier for a dataobject; identifying a device group of data storage devices across two ormore multiple-data-storage-devices enclosures, wherein the device groupis associated with the object identifier; requesting the data storagedevices in the device group to return data fragments associated with theobject identifier; asynchronously receiving at least a subset of thedata fragments from at least a subset of the data storage devices; andreconstructing the data object from the subset of the data fragments.

Asynchronously receiving at least the subset of the data fragments caninclude receiving, over a period of time, the subset of the datafragments; and completing at least another storage access operationinvolving one of the multiple-data-storage-devices enclosures during theperiod of time. Asynchronously receiving at least the subset of the datafragments can include caching each data fragment of the subset as thedata fragment is received such that reconstructing the data objectbegins when all of the subset of the data fragments have been cached.

The storage front-end system can request the data storage devices toreturn the data fragments by sending a data retrieval request to asuperset of multiple-data-storage-devices enclosures that includes thetwo or more multiple-data-storage-devices enclosures. The data retrievalrequest can be a broadcast or a multicast message. The data retrievalrequest can include a group identifier of the device group, fragmentidentifiers of the data fragments, or a combination thereof.

In various embodiments, the methods of operating the storage front-endsystem can be implemented by a computer system, e.g., the computerdevice 1200 of FIG. 12. In various embodiments, the methods of operatingthe multiple-data-storage-devices enclosure can be implemented bycontrol circuitry within the enclosure, e.g., the components describedwithin the cartridge 200 of FIG. 2.

FIG. 9 is a flow chart of a process 900 to monitor lifespan consumptionof data storage devices in a multiple-data-storage devices cartridge(“the cartridge”), consistent with various embodiments. The cartridgecan be the cartridge 100 of FIG. 1A, the cartridge 200 of FIG. 2, or themultiple-data-storage-devices enclosure 306 of FIG. 3. The process 900can be performed on the cartridge itself or via one or more storagefront-end systems (e.g., the front-end network manager 310 of FIG. 3,the front-end subsystem 402 of FIG. 4, the storage front-end system 500of FIG. 5, or the front-end subsystems 804 of FIG. 8). The cartridge canbe part of an archival storage system (e.g., the archival storage system300 of FIG. 3 or the archival storage system 400 of FIG. 4).

The process 900 includes receiving a storage access request (e.g., aread request or a write request) directed at the cartridge at block 902.Then, at block 904, the cartridge identifies which data storage devicewithin the cartridge is the target of the storage access request. Atblock 906, a lifespan consumption meter of the cartridge updates itsrecord for the identified data storage device. Each lifespan consumptionmeter can be associated with a single data storage device within thecartridge. The lifespan consumption meter increments its write accesscount when the storage access request indicates a write operation. Thelifespan consumption meter increments its read access count when thestorage access request indicates a read operation. The lifespanconsumption meter updates its record either in response to the step inblock 904 or in response to completion of the storage access operationindicated by the storage access request. The cartridge can maintain alifespan consumption meter for every data storage device in thecartridge throughout the lifetime of the cartridge or the lifetime ofthe data storage device.

The lifespan consumption meter may also include a record of how muchdata is being written (e.g., bytes or bits), how much data is being read(e.g., bytes or bits), how many times the data storage device has beenactivated (e.g., from a powered-off, spin down, or suspended state), howmany times the data storage device has been deactivated (e.g., from afully functional state), how many hours has the data storage devicebeing running on full power, how much power wattage has the data storagedevice consumed, etc. The lifespan consumption meter for each of thedata storage devices can be queried and/or reported to an administratorof the archival storage system, including the storage front-end system.Blocks 902, 904, and 906 can repeat indefinitely throughout the lifetimeof the cartridge, even after one or more of the data storage devicesfail.

At block 908, either the cartridge or the storage front-end systemcalculates a risk factor for failure, the risk factor associated witheach data storage device in the cartridge. The risk factor can be basedat least in part on the record(s) of the lifespan consumption meter. Insome embodiments, the storage front-end system can access the records ofthe lifespan consumption meter in the cartridge via a data access APIservice (e.g., the data access API service 220 of FIG. 2) of thecartridge. For example, the risk factor can be calculated as a weightedsum of the counters and records kept by the lifespan consumption meter.Different methods of calculating the risk factor can be used, includingthe methods and techniques described in U.S. patent application Ser. No.13/272,806, filed on Oct. 13, 2011.

Then, at block 910, the cartridge or the storage front-end systemdetermines whether the risk factor surpasses a threshold. In response todetermining that the risk factor surpasses the threshold, the cartridgeor the storage front-end system initiates, at block 912, a dataprotection process (e.g., migrating, replicating/copying, or other typesof backing up data) from the data storage device to another data storagedevice (the “destination data storage device”). For example, thedestination data storage device can be a data storage device with alower risk factor. The destination data storage device can be a datastorage device in another cartridge (e.g., where the cartridge isconfigured to activate only a single data storage device at a time).Particularly, the destination data storage device can be a data storagedevice within the same spin group according to the storage front-endsystem. In various embodiments, an administrator of the archival storagesystem can configure the threshold, e.g., configuring the thresholdspecific to the SLO of a particular data set (e.g., to tradeoff betweendata migration cost and probability of data loss).

The disclosed process proactively migrates data based on visible metricsavailable to the cartridge and/or the storage front-end system. Becauseof the proactive migration of data, when a failure occurs, there is ahigher probability that no data recovery procedure is necessary toservice the needs of the storage system clients.

Some embodiments involve a method of metering lifespan consumption ofdata storage devices in a multiple-data-storage-devices enclosure. Themethod can be performed by the enclosure, a storage front-end system incommunication with the enclosure, or a combination of both. The methodcan include: receiving a storage access request; identifying a firstdata storage device within the enclosure as being targeted by thestorage access request, wherein the enclosure includes two or more datastorage devices; accessing the first data storage device in response tothe storage access request; and in response to the identifying or theaccessing, updating a record of a lifespan consumption meter assigned tothe first data storage device. The lifespan consumption meter can bemaintained through a lifetime of the first data storage device or theenclosure.

The method can also include ways of calculating a risk factor based onthe lifespan consumption meter and changing the behavior of a storagesystem based on the risk factor. For example, either the enclosure orthe storage front-end system can determine a risk factor for failureassociated with the first data storage device based at least partly onthe record of the lifespan consumption meter. The enclosure can reportthe risk factor associated with the first data storage device to one ormore storage front-end systems. Such reporting can be in response toreceiving the storage access request from the storage front-end system.Alternatively, the enclosure can transmit the record of the lifespanconsumption meter to at least a storage front-end system. The lifespanconsumption meter can be maintained through a lifetime of the first datastorage device.

Based at least partly on the risk factor, the enclosure or the storagefront-end system can modify a schedule to activate or deactivate thefirst data storage device. Based at least partly on the risk factor, theenclosure or the storage front-end system can also perform a dataprotection process that involves the first data storage device when therisk factor surpasses a threshold. The data protection process can be adata migration process or a data replication process from the first datastorage device to a destination data storage device. To avoid exceedingpower limitation in the enclosure, the destination data storage devicecan be in a different enclosure.

When the storage access request is a write request, updating the recordcan include incrementing a write access counter that keeps track of howmany write accesses have been performed on or targeted toward the firstdata storage device. When the storage access request is a read request,updating the record can include incrementing a read access counter thatkeeps track of how many read accesses have been performed on or targetedtoward the first data storage device. Updating the record can alsoinclude updating a written data size counter based at least partly onhow much data is written to or is to be written to the first datastorage device in response to the storage access request. Similarly,updating the record can include updating a read data size counter basedat least partly on how much data is read from or to be read from thefirst data storage device in response to the storage access request. Theread data size counter and the write data size counter keep track of howmuch data has been read from or written to the first data storagedevice.

The lifespan consumption meter can include records related to how manytimes the first data storage device has been activated and/ordeactivated. That is, the enclosure can activate the first data storagedevice in response to or prior to receiving the storage access request.Updating the record can include incrementing an activation counter ofthe lifespan consumption meter in response to activating the first datastorage device. The enclosure can deactivate the first data storagedevice after executing the storage access request. Updating the recordcan include incrementing a deactivation counter of the lifespanconsumption meter in response to deactivating the first data storagedevice.

The lifespan consumption meter can also include records related to atotal uptime of the first data storage device. For example, theenclosure can deactivate the first data storage device after acontinuous operation of the first data storage device since theactivating of the first data storage device. Updating the record caninclude updating a total uptime counter based at least in part on a timedifference between the activating of the first data storage device andthe deactivating of the first data storage device.

Other embodiments involve a method of operating a storage front-endsystem to select which data storage device to write in based on riskfactor analysis. For example, the method can include: receiving a writerequest including payload data; generating a data fragment from thepayload data of the write request; determining risk factors associatedwith data storage devices across two or moremultiple-data-storage-devices enclosures; selecting a first data storagedevice from amongst the data storage devices based at least partly on afirst risk factor of the first data storage device; and transmitting thedata fragment to a first multiple-data-storage-device enclosure that hasthe first data storage device for storage.

The storage front-end system can determine the risk factors by receivingthe first risk factor associated with the first data storage device fromthe first multiple-data-storage-devices enclosure. That is, the firstmultiple-data-storage-devices enclosure can calculate the first riskfactor of its own data storage devices. Alternatively, the storagefront-end system can receive a failure risk-related record associatedwith the first data storage device and determine the first risk factorbased at least partly on the record. The failure risk-related record canbe a record of a lifetime consumption meter. The lifetime consumptionmeter can include records of a storage access frequency, a storageaccess count, a data throughput amount, a total uptime, or anycombination thereof, of the first data storage device. The failurerisk-related record can be a record of a fault sensor. The fault sensorcan monitor performance of the first data storage device (e.g., a writeaccess latency, a read access latency, a storage activation latency, orany combination thereof).

The storage front-end system can select the first data storage devicebased at least partly on the first risk factor as compared to other riskfactors associated with the data storage devices. Alternatively, thestorage front-end system can select the first data storage device basedat least partly on the first risk factor as compared to a threshold.

In various embodiments, the methods of operating the storage front-endsystem can be implemented by a computer system, e.g., the computerdevice 1200 of FIG. 12. In various embodiments, the methods of operatingthe multiple-data-storage-devices enclosure can be implemented bycontrol circuitry within the enclosure, e.g., the components describedwithin the cartridge 200 of FIG. 2.

FIG. 10 is a flow chart of a process 1000 for predictive fault sensingof data storage devices in a multiple-data-storage devices cartridge(“the cartridge”), consistent with various embodiments. The cartridgescan be the cartridge 100 of FIG. 1A, the cartridge 200 of FIG. 2, or themultiple-data-storage-devices enclosure 306 of FIG. 3. The process 900can be performed on the cartridge itself or via one or more storagefront-end systems (e.g., the front-end network manager 310 of FIG. 3,the front-end subsystem 402 of FIG. 4, the storage front-end system 500of FIG. 5, or the front-end subsystems 804 of FIG. 8). The cartridge canbe part of an archival storage system (e.g., the archival storage system300 of FIG. 3 or the archival storage system 400 of FIG. 4).

The process 1000 includes receiving a storage access request (e.g., awrite request or a read request) directed at the cartridge at block1002. Then, at block 1004, the cartridge executes the storage accessrequest. At block 1006, a fault sensor of the cartridge updates itsrecord for the data storage device in regards to the execution of thestorage access request.

Each fault sensor can be associated with a single data storage devicewithin the cartridge. The fault sensor measures how long it takes forthe data storage device to activate (e.g., to spin up), a read operationlatency, a write operation latency, or any combination thereof. Thefault sensor can update its record in response to the step in block1004. The cartridge can maintain a fault sensor for every data storagedevice in the cartridge throughout the lifetime of the cartridge. Thefault sensor for each of the data storage device can be queried and/orreported to an administrator of the archival storage system includingthe storage front-end system. Blocks 1002, 1004, and 1006 can repeatindefinitely throughout the lifetime of the cartridge, even after one ormore of the data storage devices fail.

At block 1008, either the cartridge or the storage front-end systemcalculates a risk factor for failure, the risk factor associated witheach data storage device in the cartridge. The risk factor can becalculated based at least in part on the record(s) of the fault sensorfor the data storage device. In some embodiments, the storage front-endsystem can access the records of the fault sensor via a data access APIservice (e.g., the data access API service 220 of FIG. 2) of thecartridge. For example, the risk factor can be calculated as a weightedsum of the counters and records kept by the fault. Different methods ofcalculating the risk factor of failure can be used similar to theprocess 900 of FIG. 9.

Then, at block 1010, the cartridge or the storage front-end systemdetermines whether the risk factor surpasses a threshold. In response todetermining that the risk factor surpasses the threshold, the cartridgeor the storage front-end system initiates, at block 1012, a dataprotection process (e.g., migrating, replicating/copying, or other typesof backing up data) from the data storage device to another data storagedevice (the “destination data storage device”). For example, thedestination data storage device can be a data storage device with alower risk factor. The destination data storage device can be a datastorage device in another cartridge (e.g., where the cartridge isconfigured to activate only a single data storage device at a time).Particularly, the destination data storage device can be a data storagedevice within the same spin group according to the storage front-endsystem. In various embodiments, an administrator of the archival storagesystem can configure the threshold, e.g., configuring the thresholdspecific to the SLO of a particular data set in terms of the tradeoffbetween data migration cost and probability of data loss.

Furthermore, the storage front-end system can actively select healthydata storage devices when writing data to the cartridge. This isaccomplished by querying the cartridge for the risk factors of its datastorage devices (e.g., through the process 900 or the process 1000) inresponse to a write request, and selecting a data storage device basedat least partly on the risk factors (e.g., selecting the data storagedevice with the lowest risk factor rating or selecting the data storagedevice that is below a threshold risk factor).

Some embodiments involve a method of fault monitoring data storagedevices in a multiple-data-storage-devices enclosure. The method can beperformed by the enclosure, a storage front-end system in communicationwith the enclosure, or a combination of both. The method can include:receiving a storage access request; identifying a first data storagedevice as being targeted by the storage access request, wherein theenclosure includes two or more data storage devices including the firstdata storage device; performing an access operation according to thestorage access request on the first data storage device; and updating arecord of a fault sensor assigned to the first data storage device bymonitoring the performing of the access operation.

The method can also include ways of calculating a risk factor based onrecords of the fault sensor and changing the behavior of a storagesystem based on the risk factor. The risk factor can be calculated andused in a similar fashion as described in this disclosure for themethods of metering lifespan consumption of data storage devices. Invarious embodiments, the records of the fault sensor or the lifespanconsumption meter can be used together or separately to calculated therisk factor of a data storage device.

When the access operation is a write operation, updating the record caninclude updating a write latency of the write operation on the firstdata storage device. When the access operation is a read operation,updating the record can include updating a read latency of the readoperation. In various embodiments, the enclosure can activate the firstdata storage device before performing the access operation. Afteractivating the first data storage device, the enclosure can update therecord of a lag time (i.e., activation latency) for the enclosure toaccess the first data storage device after power is supplied to thefirst data storage device. The record can include a moving average ofits entries (e.g., the write latency, the read latency, or theactivation latency) and/or a list of the latest measurements.

Other embodiments involve a method of a method of responding to apotential impending failure of a data storage device in amultiple-data-storage-devices enclosure. The method can be performed bythe enclosure or a storage front-end system in communication with theenclosure. For example, the method can include: determining a first riskfactor of the data storage device; determining whether the first riskfactor surpasses a threshold; selecting a destination data storagedevice from amongst data storage devices across two or moremultiple-data-storage-devices enclosures; and initiating a dataprotection process to copy data from the data storage device to thedestination data storage device.

The enclosure or the storage front-end system can determine the firstrisk factor based at least partly on performance monitoring of the datastorage device (e.g., by maintaining a record of write access latency,read access latency, activation latency, or any combination thereof, ofthe data storage device), metering operations to the data storage device(e.g., maintaining a record of a storage access frequency, a storageaccess count, a data throughput amount, a total uptime, or anycombination thereof, of the data storage device), or a combinationthereof. Alternatively, the storage front-end system can determine thefirst risk factor by receiving the first risk factor associated with thedata storage device from the multiple-data-storage-devices enclosurethat includes the data storage device.

The enclosure or the storage front-end system can select the destinationdata storage device by: selecting the destination data storage devicehaving a second risk factor that indicates a lower chance of failurethan the first risk factor; selecting the destination data storagedevice having a second risk factor that does not surpass the threshold;selecting the destination data storage device based at least partly on aranking of risk factors of the data storage devices and/or selecting thedestination data storage device that is part of a same synchronous groupof data storage devices as the data storage device. The storagefront-end system can be configured to activate and/or deactivate membersof the synchronous group together. The storage front-end system can alsobe configured to associate object identifiers of data objects maintainedby the storage front-end system with specific synchronous groups.

In various embodiments, the methods of operating the storage front-endsystem can be implemented by a computer system, e.g., the computerdevice 1200 of FIG. 12. In various embodiments, the methods of operatingthe multiple-data-storage-devices enclosure can be implemented bycontrol circuitry within the enclosure, e.g., the components describedwithin the cartridge 200 of FIG. 2.

FIG. 11 is a flow chart of a process 1100 of token-based cascade stagingof multiple-data-storage devices cartridges (the “cartridges”),consistent with various embodiments. The cartridges can each be thecartridge 100 of FIG. 1A, the cartridge 200 of FIG. 2, or themultiple-data-storage-devices enclosure 306 of FIG. 3. The cartridgescan be part of a storage rack (e.g., the rack 150). The cartridges canshare a power supply with an upper limit of how much power it cansupply. Each cartridge in the storage rack can be allocated at least afixed power usage. The process 1100 can be used to allocate additionalpower to the cartridges when powering up a data storage device withinthe cartridge or when an operation that results in a power spike isabout to happen.

The process 1100 begins at block 1102 by initiating multiple powersupply tokens in a token pool (e.g., implemented by a token buffer inmemory). The power supply tokens can retain a checked-out state or areleased state. A power supply token in the checked-out state cannot bechecked out again. Only a power supply token in the released state canbe checked out. At block 1104, a first cartridge checks out a firsttoken from the multiple power supply tokens. The first token canspecifically indicate how much power is allocated along with it.Alternatively, the first token indicates that the first cartridge canactivate one of its data storage devices. The checking out and in of thefirst token can be performed by first finding an available first tokenby sending a broadcast request, then using a two-phase commit protocol(2PC) to transfer the first token from the source. For example, thetoken pool can be maintained by one of the cartridges, be spread acrossall of the cartridges within a given power supply domain, or maintainedby an external computer system in communication with the cartridges.

Subsequently, the first cartridge activates one of its data storagedevices in block 1106. When the activation completes and a powerconsumption level of the first cartridge stabilizes to a steady-state,the first cartridge releases, at block 1108, the first token back to thetoken pool as being available.

The disclosed process enables the storage rack to better manage powersupplied to the cartridges therein. Specifically, because power spikesare associated with activation of data storage devices within thecartridges, the power supply tokens can limit the amount of data storagedevice activations at any given time. It also reduces vibration bylimiting the number of concurrent storage device activations.

Some embodiments involve a method of managing power for a firstmultiple-data-storage-devices enclosure. The method can include:checking out a first token over a network connection from a token poolshared by multiple-data-storage-devices enclosures including the firstenclosure, wherein each token of the token pool is available to bechecked out by a single device and wherein the enclosures share powerdrawn from a common power supply; after or in response to checking outthe first token, initiating activation of a data storage device withinthe first enclosure; monitoring power consumption within the firstenclosure; and releasing the first token back to the token pool when thepower consumption in the first enclosure substantially reaches asteady-state after the activation of the data storage device isinitiated. The first enclosure can be coupled to the power supply viathe network connection.

The activation of the data storage device can include supplying power instages to multiple components in the first enclosure for a processor ofthe first enclosure to establish access to the data storage device. Thefirst enclosure can release the first token when all the stages arecompleted. The activation of the data storage device can also includeproviding, by a processor in the first enclosure, access to the datastorage device to a filesystem implemented by the processor. The firstenclosure can release the first token when data in the data storagedevice is made accessible to the filesystem.

The method can further include determining that additional power isneeded beyond a power allotment granted by the first token and checkingout a second token over the network connection from the token pool whilethe first token is still checked out. The first enclosure can check outthe first token by sending a broadcast message through the networkconnection to others of the enclosures. The first enclosure can checkout the first token by sending a message through the network connectionto a known device that manages the token pool (e.g., another one of theenclosures).

Other embodiments involve a method of managing power formultiple-data-storage-devices enclosures (“the enclosures”) sharing apower supply. The method can include: maintaining a token pool shared bythe enclosures, the token pool having multiple power supply tokens eachavailable to be checked out by a single device; receiving a checkoutrequest over a network connection from a first enclosure of theenclosures; and in response to receiving the checkout request, checkingout a first token from the token pool to the first enclosure, whereinthe first token represents a unit of power under which the firstenclosure is allowed to draw from the power supply. The method canfurther include: receiving a release request from the first enclosureand, in response to receiving the release request, releasing the firsttoken into the token pool. The method can be performed by a token poolmanager (e.g., one of the enclosures or a computer system incommunication with the enclosures).

The token pool manager can initiate the multiple power supply tokensinto the token pool to be available to the enclosures. The token poolmanager can synchronize the token pool with a mirror token pool over thenetwork connection with another token pool manager.

The token pool manager can monitor power consumption drawn from thepower supply. Based at least partly on the monitoring, the token poolmanager can add or remove at least one token from the token pool. Thetoken pool manager can prevent a device that has not checked out one ofthe multiple power supply tokens from drawing power from the powersupply (e.g., by power gating).

The multiple power supply tokens can each represent an identical unit ofpower under which a device that checks out the each power supply tokenis allowed to draw from the power supply. Optionally, the multiple powersupply tokens can have at least two different subsets representingdifferent units of power under which a device that checks out a tokenfrom the each subset is allowed to draw from the power supply.

In various embodiments, the methods of operating themultiple-data-storage-devices enclosure can be implemented by controlcircuitry within the enclosure, e.g., the components described withinthe cartridge 200 of FIG. 2.

While processes or blocks are presented in a given order in FIGS. 4-11,alternative embodiments may perform routines having steps, or employsystems having blocks, in a different order, and some processes orblocks may be deleted, moved, added, subdivided, combined, and/ormodified to provide alternative or subcombinations. Each of theseprocesses or blocks may be implemented in a variety of different ways.In addition, while processes or blocks are at times shown as beingperformed in series, these processes or blocks may instead be performedin parallel, or may be performed at different times.

It is understood that the use of relational terms, if any, such as firstand second, and the like are used solely for distinguishing one entityor action from another, without necessarily requiring or implying anysuch actual relationship or order between such entities or actions.

FIG. 12 is a block diagram illustrating a diagrammatic representation ofa computing device 1200 within which a set of instructions, for causingthe machine to perform any one or more of the methodologies or modulesdiscussed herein, may be executed. For example, the computing device1200 can be part of the cartridge 100 of FIG. 1A, the cartridge 200 ofFIG. 2, the data range processor module 302 or the front-end networkmanager 310 of FIG. 3, the front-end subsystem 402 or one of the storagesubsystems 404 of FIG. 4, the storage front-end system 500 of FIG. 5,one of the clients 802, the front-end subsystems 804 or the cartridges810 of FIG. 8, or any combination thereof, In alternative embodiments,the machine may comprise or include a network router, a network switch,a network bridge, personal digital assistant (PDA), a cellulartelephone, a Web appliance or any machine capable of executing ortransmitting a sequence of instructions that specify actions to betaken. The computing device 1200 is intended to illustrate a hardwaredevice on which any of the instructions, processes, modules andcomponents depicted in the figures above (and any other processes,techniques, modules and/or components described in this specification)can be implemented. As shown, the computing device 1200 includes aprocessor 1202, memory 1204, non-volatile memory 1206, and a networkinterface 1208. Various common components (e.g., cache memory) areomitted for illustrative simplicity. The computing device 1200 can be ofany applicable known or convenient type, e.g., a personal computer (PC),server-class computer or mobile device (e.g., smartphone, card reader,tablet computer, etc.). The components of the computing device 1200 canbe coupled together via a bus and/or through any other known orconvenient form of interconnect.

One of ordinary skill in the relevant art will recognize that the terms“machine-readable (storage) medium” or “computer-readable (storage)medium” include any type of device that is accessible by the processor1202. The memory 1204 is coupled to the processor 1202 by, for example,a bus 1210. The memory 1204 can include, by way of example but notlimitation, random access memory (RAM), e.g., dynamic RAM (DRAM) andstatic RAM (SRAM). The memory 1204 can be local, remote, or distributed.

The bus 1210 also couples the processor 1202 to the non-volatile memory1206 and drive unit 1212. The non-volatile memory 1206 may be a harddisk, a magnetic-optical disk, an optical disk, a read-only memory(ROM), e.g., a CD-ROM, Erasable Programmable Read-Only Memory (EPROM),or Electrically Erasable Programmable Read-Only Memory (EEPROM), amagnetic or optical card, or another form of storage for large amountsof data. The non-volatile memory 1206 can be local, remote, ordistributed.

The data structures, modules, and instruction steps described in thefigures above may be stored in the non-volatile memory 1206, the driveunit 1212, or the memory 1204. The processor 1202 may execute one ormore of the modules stored in the memory components.

The bus 1210 also couples the processor 1202 to the network interface1208. The network interface 1208 can include one or more of a modem ornetwork interface. A modem or network interface can be considered to bepart of the computing device 1200. The network interface 1208 caninclude an analog modem, ISDN modem, cable modem, token ring interface,satellite transmission interface (e.g. “direct PC”), or other interfacesfor coupling a computer system to other computer systems.

It is to be understood that embodiments may be used as or to supportsoftware programs or software modules executed upon some form ofprocessing core (e.g., the CPU of a computer) or otherwise implementedor realized upon or within a machine or computer-readable medium. Amachine-readable medium includes any mechanism for storing ortransmitting information in a form readable by a machine, e.g., acomputer. For example, a machine-readable medium includes read-onlymemory (ROM); random access memory (RAM); magnetic disk storage media;optical storage media; flash memory devices; electrical, optical,acoustical or other form of propagated signals, for example, carrierwaves, infrared signals, digital signals, etc.; or any other type ofmedia suitable for storing or transmitting information.

This disclosure includes other aspects, elements, features, and steps inaddition to or in place of what is described in the figures. Thesepotential additions and replacements are described throughout thespecification. For example, some embodiments involve a method of writingdata through a data range application programming interface (“API”)implemented on a multiple-data-storage-devices cartridge (the“cartridge”). The method includes receiving a write request from arequester device, wherein the write request is a direct, broadcast ormulticast, or fanout message and includes a size indication for acontiguous range of data; responsive to receiving the write request,sending a response message to the requester device indicating an intentto store the contiguous range of data; receiving the contiguous range ofdata from the requester device; powering on a target data storage devicefrom amongst data storage devices within the cartridge while keeping atleast another data storage device in the cartridge powered down; andwriting the contiguous range of data to the target data storage device.

In response to receiving the write request, the cartridge can select thetarget data storage device from amongst the data storage devices withinthe cartridge. The cartridge can send the response message with astorage device identifier of the target data storage device to therequester device. The cartridge can power on the target data storagedevice in response to sending the response message. That is, unless thetarget data storage device is already powered on.

The write request can indicate that the target data storage device isselected by the requester device to store the contiguous range of data.The requester device can be a storage front-end system or a clientdevice. Powering on the target data storage device can be responsive toreceiving the write request that indicates the target data storagedevice. That is, unless the target data storage device is alreadypowered on. The cartridge can send failure risk factors for each of thedata storage devices within the cartridge to the requester device aspart of the response message. The cartridge can also specifically send afailure risk factor for the target data storage device to the requesterdevice as part of the response message when the target data storagedevice is already identified by the write request.

The cartridge can write the contiguous range of data by writing to thetarget data storage device sequentially without returning to apreviously written memory section in the target data storage devicewhile the target data storage device has been continuously powered on.The cartridge can write the contiguous range of data without modifyingdata already written in the target data storage device.

Some embodiments involve a method of reading data through a data rangeAPI implemented on a multiple-data-storage-devices cartridge. The methodof reading data can include: receiving a read request from a requesterdevice, wherein the read request is a direct, broadcast or multicast, orfanout message and includes a data range identifier associated with acontiguous range of data; responsive to receiving the read request,determining whether the contiguous range of data is found in thecartridge and which target data storage device from amongst data storagedevices within the cartridge contains the contiguous range of data;powering on the target data storage while keeping at least another datastorage device in the cartridge powered down; retrieving the contiguousrange of data from the target data storage device; and sending thecontiguous range of data to the requester device.

The data range identifier can include a spin group identifier associatedwith member data storage devices across different cartridges includingthe cartridge that received the read request. The contiguous range ofdata is a data fragment associated with other data fragments in othermember data storage devices across the different cartridges. The readrequest may include a group identifier of a spin group such that thecartridge can determine whether the contiguous range of data is found inthe cartridge by determining whether one of the data storage deviceswithin the cartridge is in the spin group. The cartridge can alsodetermine which target data storage device contains the contiguous rangeof data by determining which target data storage device is in the spingroup.

Some other embodiments involve a multiple-data-storage-devicescartridge. The cartridge can include: two or more data storage devices;a data communication port to receive a data connection to a computernetwork and to receive and supply power to themultiple-data-storage-devices cartridge; a power management circuitryfor managing power supplied from the data communication port to one ormore of the data storage devices; and control circuitry comprisingmemory storing executable instructions and a computing component. Thecomputing component can be configured by the executable instructions toexecute any of the methods for reading and writing data described above.

What is claimed is:
 1. A method of operating an enclosure comprising:based on receiving a first read request for a first data fragment,identifying a first target storage device of a first subset of storagedevices as storing the first data fragment based, at least in part, onthe first read request; determining whether the first subset of storagedevices is active based, at least in part, on a cyclical activationschedule; and based on determining that the first subset of storagedevices is not active based, at least in part, on the cyclicalactivation schedule, indicating the first read request in a first readrequest queue for the first subset of storage devices; based onindicating the first read request in the first read request queue,determining a number of read requests in the first read request queueand a number of read requests in a second read request queuecorresponding to a second subset of storage devices to determine whetherthe cyclical activation schedule should be modified to prioritize thefirst subset of storage devices based, at least in part, on thedetermined numbers of read requests; and in association with activationof the first subset of storage devices in accordance with the cyclicalactivation schedule, removing the first read request from the first readrequest queue; and retrieving the first data fragment from the firsttarget storage device.
 2. The method of claim 1, wherein the first readrequest includes a storage device identifier of the first target storagedevice, wherein identifying the first target storage device is based onthe storage device identifier.
 3. The method of claim 1, wherein thefirst read request is broadcast to a plurality of enclosures comprisingthe enclosure.
 4. The method of claim 1, wherein the first read requestincludes an identifier of the first data fragment, wherein theidentifier of the first data fragment comprises a fragment identifier oran object identifier of a data object that originated the first datafragment; and wherein identifying the first target storage device isbased on the identifier and a mapping of the identifier to the firsttarget storage device stored in a memory of the enclosure.
 5. The methodof claim 1, wherein the first read request includes a group identifierassociated with each storage device in the first subset of storagedevices, wherein identifying the first target storage device is based,at least in part, on the association of the group identifier with thefirst target storage device.
 6. The method of claim 1, furthercomprising: based on receiving a second read request for a second datafragment after receiving the first read request, identifying a secondtarget storage device storing the second data fragment based, at leastin part, on the second read request, wherein the second subset ofstorage devices comprises the second target storage device; and based ondetermining that the second subset of storage devices is active inaccordance with the cyclical activation schedule, retrieving the seconddata fragment from the second target storage device prior to retrievingthe first data fragment.
 7. The method of claim 1, wherein the cyclicalactivation schedule is based, at least in part, on health statistics ofthe first subset of storage devices and the second subset of storagedevices.
 8. The method of claim 1, further comprising: determining afirst risk factor for the first subset of storage devices and a secondrisk factor for the second subset of storage devices, wherein a riskfactor indicates a likelihood that a storage device in a subset ofstorage devices will fail; and modifying the cyclical activationschedule based, at least in part, on the first risk factor and thesecond risk factor.
 9. The method of claim 8, wherein modifying thecyclical activation schedule comprises reducing an activation durationof the first subset of storage devices based, at least in part, on thefirst risk factor exceeding a threshold.
 10. An enclosure comprising: aprocessor; and a machine-readable medium comprising program codeexecutable by the processor to cause the enclosure to, based onreceiving a first read request for a first data fragment, identify afirst target storage device of a first subset of storage devices asstoring the first data fragment based, at least in part, on the firstread request; determine whether the first subset of storage devices isactive based, at least in part, on a cyclical activation schedule; andbased on a determination that the first subset of storage devices is notactive based, at least in part, on the cyclical activation schedule,indicate the first read request in a first read request queue for thefirst subset of storage devices; based on indication of the first readrequest in the first read request queue, determine a number of readrequests in the first read request queue and a number of read requestsin a second read request queue corresponding to a second subset ofstorage devices to determine whether the cyclical activation scheduleshould be modified to prioritize the first subset of storage devicesbased, at least in part, on the determined numbers of read requests; andbased on activation of the first subset of storage devices, remove thefirst read request from the first read request queue; and retrieve thefirst data fragment from the first target storage device.
 11. Theenclosure of claim 10, wherein the cyclical activation schedule is basedat least in part, on health statistics of the first subset of storagedevices and the second subset of storage devices.
 12. The enclosure ofclaim 10, wherein the machine-readable medium further comprises programcode executable by the processor to cause the enclosure to: based onreceiving a second read request for a second data fragment afterreceiving the first read request, identify a second target storagedevice storing the second data fragment based, at least in part, on thesecond read request, wherein the second subset of storage devicescomprises the second target storage device; and based on determiningthat the second subset of storage devices is active in accordance withthe cyclical activation schedule, retrieve the second data fragment fromthe second target storage device prior to retrieving the first datafragment.
 13. One or more non-transitory machine-readable media havingprogram code for asynchronously servicing a read request stored therein,the program code to: based on receiving a first read request for a firstdata fragment, identify a first target storage device of a first subsetof storage devices as storing the first data fragment based, at least inpart, on the first read request; determine whether the first subset ofstorage devices is active based, at least in part, on a cyclicalactivation schedule; and based on a determination that the first subsetof storage devices is not active based, at least in part, on thecyclical activation schedule, indicate the first read request in a firstread request queue for the first subset of storage devices; based onindication of the first read request in the first read request queue,determine a number of read requests in the first read request queue anda number of read requests in a second read request queue correspondingto a second subset of storage devices to determine whether the cyclicalactivation schedule should be modified to prioritize the first subset ofstorage devices based, at least in part, on the determined numbers ofread requests; and based on activation of the first subset of storagedevices, remove the first read request from the first read requestqueue; and retrieve the first data fragment from the first targetstorage device.
 14. The machine-readable media of claim 13, furthercomprising program code to: based on receiving a second read request fora second data fragment after receiving the first read request, identifya second target storage device storing the second data fragment based,at least in part, on the second read request, wherein the second subsetof storage devices comprises the second target storage device; and basedon determining that the second subset of storage devices is active inaccordance with the cyclical activation schedule, retrieve the seconddata fragment from the second target storage device prior to retrievingthe first data fragment.